General representation of collective neural dynamics 
with columnar modularity 
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We exhibit a mathematical framework to represent the neural process at the cortical level. The 
description of neural dynamics with columnar and functional modularity, named the fibre bundle 
map (FBM) representation method, is based on both neuroscience and informatics, whereas it leads 
to the conventional formulas in statistical physics. The possibility of analogy between the phenomena 
in brain and physical systems has been proposed jjIQ. In spite of the complex circuitry and the 
nonlinear dynamics in neural systems, the neural behavior at high levels may be described by simple 
and general rules, which related to the noble theory in statistical physics. The FBM method gives 
profit in building or analyzing the neural models by representing essential ingredients of neural 
interactions by general formulas. We apply our method to the proposed models of visual map 
formation and show how they can share statistical properties with vortex dynamics in magnetism 
in spite of various development mechanisms. 

PACS numbers: 87.10.+e,87.19.La,89.75.Fb 
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I. INTRODUCTION 

Though the detailed dynamics of a single neuron are 
revealed, there still remains a challenge at the network 
level to explain how brains perform higher cognitive func- 
tions. Studies of physical models are focused on achieving 
a biological realism of the neural computation models. 
However, the success of the basic neural network mod- 
els, based on the connectional framework between simple 
cells, in the application of small adaptive systems, they 
are inherently problematic in the apprehension of col- 
lective neural phenomena and higher cognitive behavior 
in the real brain. And also there are attempts to see 
through the neural processing at higher levels, the func- 
tional modularity of neurons or the symbolic processing 
architecture. Before the physiological evidence of repet- 
itive cortical blocks, there were proposals of the modu- 
larity within neighbor neurons, the called cell assemblies 
(CAs), considering the high dimensional attribute and 
faculty of neurons It is a tendency of neurons to ag- 
gregate together with similar functional specializations 
and make organizations hierarchically. Though different 
classifications and names for neural clusters, we adopt 
the suggestion that neuron - minicolumn - (hypercolumn) 
- macrocolumn - cortex area - hemisphere, where mini- 
column is a candidate for "the repeating pattern of cir- 
cuitry" or "the iterated modular unit" [4|. 

In this paper, we exhibit a mathematical framework, 
noted briefly and named the fibre bundle map (FBM) 
methods in ref. and show how to represent the neu- 
ral process generally at the cortical level. Briefly speak- 
ing, the FBM representation is a mapping of the feature 



components (often the synaptic weights) to topological 
spaces, the called bundles. The pattern informations 
are reordered in locally transformed coordinates and a 
few of major components are extracted. Obviously there 
exist another mathematical framework to represent the 
neural process in reduced space. Kohonen set up the 
mathematical preliminaries, the called feature maps or 
feature-based representation, in vector space and led the 
successive models in artificial and physiologic neural net- 
works . Symbolic processing architectures also suggest 
the description of neural computations at the cognitive 
and rational bands. The feature vector space or the sym- 
bolic sets can belong to a kind of FBM representation. 
But the FBM method has an interest in the manifold 
structure of frequent inputs in feature space and its cor- 
responding symmetry group. Indeed, the properties of 
neural progress are governed not by the detailed neural 
interaction rules but by the algebraic structure of domi- 
nant feature components. The FBM method represents 
the neural process by the general formulas in statistical 
physics and helps to comprehend collective neural phe- 
nomena intuitively through the knowledge in statistical 
mechanics and differential geometry. 

As a class of abstract representations, the formulas in 
the FBM models have some different character with those 
in the feature-based (often the called "low-dimensional" 
feature vector) models. In the feature-based representa- 
tion, the change in the feature vector at position r, $(r) 
is described as the difference in the stimuli vector $(r'), 
such as A$(r) cx ($(r') — 3>(r)) with the energy of the 
form |$(r) — <&(r')| 2 (or its higher powers). Whereas, in 
the FBM representation, the interactions between neu- 
rons are notated by inner products rather than their dis- 
tance, and it is generally assumed that the energy of neu- 
ral process can be expanded in a power series, i.e., 
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where the field variables i/j(r) denote the feature state of 
neurons at cortical location r. We will show how this 
formula is derived and the interaction functions are de- 
termined in the simple (or the called "high-dimensional" 
feature vector representation) and the complex cell mod- 
els. In a continuum limit, the energy can be approxi- 
mated to 

Sty] = Jdr |||(V - iA)Vf + ^-\M 2 + f; H 4 } , (2) 

where the odd power terms are expected to be vanished 
generally. This is just the Ginzburg-Landau energy with 
gauge invariance and explains the statistical properties 
of the emergent cortical maps in experiments and simu- 
lations. The energy in a continuum approximation often 
can be derived using only minimal mathematical con- 
straint such as the symmetry. The energy form in Eq. 
and Eq.@ proposes the possibility of the analogy be- 
tween the physical and the neural systems, and the char- 
acteristics of developed visual maps are systematically 
apprehended through the statistical properties of vortices 
in magnetism Q, Qj. 

We apply the FBM representation method to the de- 
velopment models in visual cortex. The cortical map 
formation in orientation and ocular dominance columns 
is one of the most studied problems in brain. A consider- 
able amount of different models is proposed, and some of 
which arc compared with the experimental findings and 
in competition • Miller et al. formulated correlation- 
based models describing how ocular dominance and ori- 
entation columns develop in simple cell models @, 0, [ljj . 
Obermayer et al. presented a statistical-mechanical anal- 
ysis of pattern formation and compared predictions quan- 
titatively with experimental data using the Kohonen's 
self-organizing feature map (SOFM) approaches. Wolf et 
al. obtained again the conditions for the emergence of 
a columnar pattern in the SOFM algorithm 11]. The 
studies of the elastic-net model also show the bifurca- 
tion and emergence of a columnar pattern [T2I ITsl IT3 ] . 
Scherf et al. investigated the pattern formation in ocu- 
lar dominance columns with more detailed model, which 
covers the results of the SOFM algorithm and the elastic- 
net model 0. Wolf and Geisel predicted the influence 
of the interactions between ocular dominance and orien- 
tation columns on the pinwheel stability without model 
dependency and demonstrated it in the simulations of the 
elastic-net model 0]. The lateral (or neighbor) interac- 
tion models are also successful scheme based on physiol- 
ogy HHHHIIl 

In the proposed visual map formation models, the 
Hamiltonian models with spin variables belong to the 
class of FBM representation models P,0,H3]- Other de- 
velopment models written in the high- or low-dimensional 



feature vector representation can be revised again in the 
FBM representation. The formulas in FBM models rep- 
resent essential ingredients of neural interactions with- 
out paying much attention to particular neural control 
mechanism. Moreover, the modification of the iterative 
procedure of a model into the formula in Eq. or Eq. J2J 
becomes the statistical analysis of the model itself. The 
quadratic interaction function D(vi,rj) is consequence 
in the visual map formation as other physical systems. 
The interaction functions in neural process mean more 
than the intracortical connections or recurrents in the 
called lateral activity control. In the competitive Heb- 
bian models, such as the elastic-net model and the SOFM 
algorithm, the interaction functions comprise the feedfor- 
ward competition or normalization process. However, in 
the FBM representation the functional matrix D(ri,Tj) 
of the visual map formation models have common shape, 
the called Mexican hat type, that is, positive in short- 
range and negative in long-range, in spite of different 
development mechanisms. The bifurcation to a inhomo- 
geneous state and the emergence of a columnar pattern 
is possible when there are strong negative interactions 
in D(vi,Yj). The development of a columnar pattern is 
also concerned with non- vanishing vector A, the called 
vector potential in physics, in Eq.(J2Jl. The FBM repre- 
sentation method will show how the development models 
with different mechanisms lead to the successful forma- 
tion of visual maps and share the statistical properties of 
vortices in the spin Hamiltonian models. 



II. REPRESENTATION OF NEURAL STATE 
WITH COLUMNAR MODULARITY 

The structures and connections in cerebral cortex are 
more complex and modular than those in artificial neu- 
ral networks. Neurons tend to be vertically arrayed in 
the cortex, forming cylinders known as cortical columns. 
Traditionally, six vertical layers have been distinguished 
and classified into three different functional types. The 
layer IV neurons (IN box), first get the long-range in- 
put currents, and send them up vertically to layer II and 
III (INTERNAL box) that are the called true associa- 
tion cortex. Output signals are sent down to the layer V 
and VI (OUT 'box) , and sent further to the thalamus or 
other deep and distant neural structures. Lateral connec- 
tions also occur in the superficial (layer II and III) pyra- 
midal neurons. In columnar (or horizontal) clustering, 
there are minicolumns, which are consisted of about 100 
neurons and 30 um in diameter in monkeys, and macro- 
columns, which are 0.4~1.0 mm and contain at most a 
few hundred minicolumns. On the wider discrimination, 
there are 52 cortex areas in each human hemisphere; a 
Brodmann area averages 21 cm 2 and 250 million neurons 
grouped into several million minicolumns 4]. 

The columnar modules can be regarded as a kind of 
multi-layered neural networks and would have complex 
functional attributes. Most neurons in brain have the 
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attribute of selective response to a received activity, and 
the preferred signals become an useful representation of 
the functional attributes in a small neural region. A tra- 
ditional representation of neural state is the vector nota- 
tion v, where its components correspond to the activity 
of each neuron in receptor layer. If a columnar module 
(or complex cell) at position r respond selectively to a 
particular input vector v and make an output vector y, 
its functional attribute can be represented compactly as, 

w(r) = y o F o v T , (3) 

where F is the nonlinear response or activation function 
of complex cell. If the activation function is linear or ig- 
nored, this leads to a simple pattern associator, the called 
linear associator. The experiments of the response prop- 
erties to external stimuli through electrode penetration 
can be understood as the measurement of the product 
between the associator w(r) and the input signal v' : 

Kr)ov'| = |y|F(v T v'), (4) 

where the activity of the output |y| corresponds to the 
measurement of the number of action potentials or the 
frequency of spikes. In the physiological experiments 
with the complex cells in primary visual cortex 12 ll or 
the object perceptions in inferotemporal (IT) cortex |22| . 
the response property of columnar modules used to be the 
combination of different patterns and then the functional 
form in Eq.JSJ would be expanded into the summation 
of associators. When the output y is common with the 
most favorite input v such as Hopfield networks |23j or 
the most favorite input is only concerned, a vector no- 
tation can play the role of representation of functional 
attributes in columnar modules. 

FigD depicts a neural network with columnar mod- 
ules. A matrix W denotes the feedforward synaptic 
weights cross cortex areas, such as the connections be- 
tween LGN and primary visual cortex, and the input 
vector to a columnar module is given by = W^u (or 
v i = WiaUa). In a complex cell model, it is expected 
that the synaptic connections within a columnar module 
w(r) achieve the functional attributes of neuron, whereas 
in a simple cell model, the connections with the external 
cells W are considered to vest the functional attributes. 
For example, the ocular dominance in primary visual cor- 
tex is determined whether a neuron in VI is more con- 
nected to the left or right eye (or LGN) cells. We call 
this the extrinsic information coding type, which is real- 
ized by the connectivity of far neurons cross cortex ar- 
eas, whereas the intrinsic type is realized by the synaptic 
plasticity between close neurons within a columnar mod- 
ule. The neural attribute of two coding types are rep- 
resented by common formula in FBM models, but there 
exist some different ground when building actual models. 
The feedforward competition behavior should be related 
to the intrinsic coding type. Moreover, the extrinsic en- 
coding type causes a problem in modeling huge networks 
because too massive connections are required when the 
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FIG. 1: A neural network model with columnar modules 
with function tu'*'. Input signal to a columnar module Vi 
is driven by feedforward synapses with weights W, that is, 
v, = WiU, and its output y, is interconnected to neighbors 
by intracortical connections J. 



meaning of activity is characterized only from where the 
current come. We expect that the intrinsic type, encod- 
ing information in spatial or temporal correlations within 
a signal band, is essential in huge networks and would be 
a prominent strategy in the real brain. 



III. FIBRE BUNDLE MAP REPRESENTATION 

The FBM representation method bases on a mathe- 
matical framework - the called fibre bundle in manifold 
theory p4l . Eflj . For a trivial fibre bundle, a total (or 
bundle) space E, which will depicts the neural attributes 
at a cortical area, is composed of a base space B and 
a fibre F, tat is, E = B x F. In our interests, cortical 
locations are the elements in base space, where feature 
(often pattern, code or model) space becomes a fibre. A 
structure (or symmetry) group G is a homeomorphism of 
fibre F, and the same with the fibre F in a principal fibre 
bundle. The principal fibre bundles admit connexions (or 
vector potentials in physics), and it is for this reason that 
they are of basic importance in gauge theories in physics. 
The features of cortical cells or small cortical regions at 
each cortical location r are represented by a set of field 
variables ipa( r ) and 

V(r) = |V(r)| exp(-^ a (r)r Q ) = ^ a (r)r a , (5) 

where <^ a (r) is an arbitrary internal (feature) phase and 
r a is the basis of a continuous (or Lie) group G. The 
bases can be taken as the amount of receptor cells, but 
are usually reduced according to the statistical structure 
of inputs. The frequent inputs usually occupy small re- 
gions in the total feature space and the major variance of 
feature components occurs within a embedded subman- 
ifold with high stimuli density (Fig. |2J) The reduction of 
feature space is related to the extraction of features from 
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FIG. 2: Probabilistic external stimuli and a potential func- 
tion with external source. The transformed basis r and r 
are the principal directions of external stimuli density at a 
point. 

inputs in learning rules as well. Symmetry breaking be- 
tween transformed feature components is expected in the 
neural progress of experience and learning, and cortical 
dynamics can be described with a few of field components 
in a reduced feature space. 

The differential geometric concepts in the FBM repre- 
sentation furnish an intuitive explanation for emergent 
cortical maps. The self-organization of feature maps 
achieved by locally gathering similar interests means 
there is smooth variance of features with neighbor neu- 
rons at each location. In other words, the properties 
of "organized" and "optimized" feature maps is related 
with those of "continuous" and "flat" variables in man- 
ifold. If there is no difference of features with neighbors 
at small region near position r, they can be denoted by 
V^(r) = (or V0(r) = 0). If there exists small tilting 
of phase angle at position r and an arbitrary vector A(r) 
denote the difference between phase angles, the called co- 
variant derivative is given by (V — iA(r))-0(r) = (or 
V0(r) — A(r) =0). If the covariant derivative vanishes 
(said to be flat or parallel by translated in manifold the- 
ory) for all r, the distribution of the field variables ^(r) 
would be a minimum solution of the integral 

S = j air |(V-iAM 2 , (6) 

for the connexion A. A non-vanishing connexion A oc- 
curs when there are strong competitive behavior or in- 
hibitory lateral interactions between neurons, and is re- 
lated to the emergence of a periodic pattern in cortical 
maps, such as the band patterns in ocular dominance 
columns and the linear zones in orientation preference 



FIG. 3: The simulation result of orientation map formation. 
The orientation maps have U(l) (or 0(2)) symmetry and the 
major characteristics of the developed map can be predicted 
using only the symmetry properties. 




FIG. 4: The complete pattern of ocular dominance stripes 
in the striate cortex of a macaque monkey. There is a strong 
tendency for the stripes to meet the margin of striate cortex at 
steep or right angles. (Reprinted by permission from S.LeVay, 
Copyright ©1985 by the Society for Neuroscience |2fi|.l 



columns, with the wavelength A = 27r/|A|. Fig0]shows 
the complete pattern of ocular dominance stripes of a 
macaque monkey. The orthogonality between the con- 
tour lines of feature map and the boundary of cortical 
area is a property of minimal solutions in Eq.©. From 
the condition SS/S(j) - or V 2 - for tp = e 2i<t > with 
the preferred angle <fi, the normal component of V</> van- 
ishes at the area boundary since the integral along a nar- 
row rectangular loop over the area boundary § c V0 • dn 
vanishes due to the divergence theorem. Such perpen- 
dicularity with the area boundary is also manifested in 
other static field solutions, such as the magnetic field. 

The symmetry property also helps to predict the en- 
ergy function of the cortical map formation. For exam- 
ple, the features of orientation preference columns in the 
visual cortex have U(l) (or 0(2)) symmetry. However 
we perform a rotation in all the preferred angles through 
same angle (<j) — ► <j) + x - called 'global' gauge transform), 
the energy of orientation map formation should remain 
invariant. The rotation angle \ can have a dependency 
on position r, the called 'local' gauge transform, and the 
energy in a continuum approximation may take the form 
in Eq.@ or Eq.© with A = V%(r). 
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IV. DESCRIPTION OF DETAILED NEURAL 
INTERACTIONS 

The description of neural dynamic at a high level also 
should be based both on neuroscience and informatics. 
One important principle for neural plasticity is the Heb- 
bian rule : two simultaneously active neurons on either 
side of a connection increases the weight of that connec- 
tion is increased 3]. The simple Hebbian plasticity rule 
in a single neuron consists of inputs u and weights W 
takes the form 



AW(<) oc y(t)u(t) 



(7) 



for the output y = /(Wu) with the activation function 
/ of simple cell. In intracortical connected networks, the 
input becomes the summation of the current from input 
and neighbor cells. The output of neuron at i-th site 
becomes 



V, 



Vi) 



(8) 



for Vi = Wi a u a and the recurrent weight matrix J. In a 
energy model, synaptic plasticity rule is regarded as the 
negative gradient of an energy (often objective, error or 
cost function) defined as a function of W : 



AW 



dE[W] 
dW ' 



(9) 



Because of the nonlinearity of the activation function and 
the recursive form in Eq.JHJ), the energy used to be ap- 
proximated depending on models. For example, with the 
assumption of y,; = /(i>i + J2j Jij v j) an d a series expres- 
sion of activation function f(v) — J^tW+i.t' , the energy 
is obtained by 



E l W ] ^Et^C^^c.:-^. (10) 



where 



D 



D 



(2) 



J, 

■ D 



H»2 / 

(2) 



Ji, 



(11) 



In the FBM representation of simple cell model, feedfor- 
ward synaptic weights Wi Q is replaced as field variables 
tpa{^i), then the energy in EcL lfTH)) satisfies the form of en- 
ergy in Eq.QJ. For efficient description of dynamics, the 
energy is decomposed into the functions of transformed 
field variables. Because of the anisotropy in input cor- 
relation Q (often in neighbor activity D), the symmetry 
between components is broken and the effective dynam- 
ics can be described with a few of dominant components. 
The consequence of the anisotropy in neighbor activity 
between feature components is explored in the case of 
the anisotropy between orientation and ocular dominance 
columns @|. 

In a complex cell model, the features of neurons relate 
to the synapses within a columnar module. The colum- 
nar module is a kind of adaptive neural network systems 
and the modulation of its functional attributes involves 
intricate changes in synaptic weights. An effective as- 
sumption is that the output of a columnar module is one 
of the proper states of the functional and will change 
following afferent signals. For the currents from input 
and neighbor cells and a linear activation function, the 
change in the proper state or the output of a columnar 
module is then 



Ay, + J ijyj 



(14) 



for the input Vj to the columnar module at position rj 
and the energy averaged over inputs is obtained by 

E iy] = - I>i>i>y< - \ E J av iYi- (1 5 ) 

i i,j 

In the FBM representation, the output with multivari- 
able is replaced by field variables : 



or 



£M=-^%)^( ri ) (17) 

i 



is the functional tensor of rank £ and 



(12) 



is the input correlation tensor of rank £. ( ■ }v denotes 
the average over input data set T>. This energy based 
on the basic Hebbian rule is adjusted again depending 
on the characteristic of synaptic plasticity rules j27| ■ For 
example, the covariance plasticity rule replaces the in- 
put correlation function Q^' with rank £ as the input 
covariance function 



((u ai - (u ai )v) ■ ■ ■ (u ae - (u ae )v))v- (13) 



where a functional vector B(r^) = (vi)x> is the linear 
average over inputs. If we assume ip' and ip are creation 
and annihilation operators, the term ^(r^) J(rj, Tj)ip'(rj) 
can be regarded as the description of phenomena that a 
created activity at position rj is translated with kernel J 
and annihilated at position r^. 

A series of physiological experiments showed that the 
synaptic plasticity comes from a redistribution of the 
available synaptic efficacy, not an increase in the effi- 
cacy (2?],|2g. In other words, the neural plasticity at the 
network level can be understood as the pursuit of incre- 
ment in the probability of reactivity with bounded total 



synaptic strength for environmental experience. With 
the expectation of a automatic normalization of synap- 
tic weights, the norm of field variables \tp\ used to be 
constrained to be constant. In this sense, the neural dy- 
namics with functional modularity may be described by 
the slight shift in the internal phase per activity following 
afferent signals. Sometimes the normalization constraint 
is not imposed and involved in the plasticity rule with 
subtractive normalization |29j| . For the energy function 
of the form 

E[tp] = at/j 2 - fo/A (18) 

the stability of synaptic weight can be achieved due to 
the relaxation of to its equilibrium value. 

Another important mechanism expected in neural 
computation is the enhancement of neural activity de- 
pending on correspondence to input. A possible enhance- 
ment modulation is the restriction on the sum over the 
activity by subtractive normalization. With a simple 
nonlinear form x + r\x 2 , the external source term with 
enhanced afferent signals becomes that 

= \>/.)E^(i + W/ 

^ (vi)x>V>i + - ^ Sijipiipj (19) 

3 

for Vj = p(v and p = J^. pi with the stimuli strength pi 
at position rj. The scattering function with a input data 
set T> is defined as 

Sij = 2r ] (v i v j ) v (S ij - 1) (20) 

for the enhancement (or competition) parameter rj. In 
the FBM representation, the scattering function de- 
scribes the feedforward competition process in the com- 
petitive Hamiltonian models, such as the elastic-net 
model and the SOFM algorithm. In fact, for an intrinsic 
coding type, network cannot tell which neurons match 
mostly with input signal a priori and the winner has to 
be determined after lateral inhibitory activity. The com- 
petitive Hebbian models require a normalization control 
of response or a priori decision of winner and depict the 
feature vectors in visual cortex through the connectiv- 
ity between visual cortex and retinas (or LGNs) [l5| . 
The lateral activity function J(ri,rj), the connectivity 
between neurons (or columnar modules) at position r.; 
and Tj within a cortex area, has two types according to 
the control mechanisms 30]. In the case of the lateral 
feedback control (which Kohonen called the activity-to- 
activity kernel), the lateral activity function J is regarded 
to be excitatory for short distance and inhibitory for long 
distance with the so-called Mexican hat type (Fig|SJi). 
Whereas in the case of the lateral control of plasticity (or 
the activity-to-plasticity kernel), the lateral interaction is 
nonnegative and may take the Gaussian form (FiglSJ)). 
The competitive Hebbian models take the lateral con- 
trol of plasticity, that means there is no negative value 



G 



J(r)j J(r) 




(a) Lateral feedback control (b) Lateral control of 
of activity plasticity 



FIG. 5: The two types of neighbor interaction functions and 
control mechanisms, (a) The lateral interaction models adopt 
lateral activity control and the activation kernel, usually so- 
called "Mexican hat" function (positive feedback for close dis- 
tance and negative for longer distance), (b) The plasticity 
control with nonnegative kernel requires feedforward competi- 
tion (or feedforward normalization of activity over networks) . 
The elastic-net model assumes the nearest neighbor interac- 
tions (or elastic force), whereas the SOFM algorithm takes 
the neighbor function on Gaussian form with the hard com- 
petition (or winner-take-all activity). 



in J, and the scattering function S from afferent signal 
enhancement has an effect of inhibitory activity. 

Now we employ the concepts of thermodynamic into 
neural dynamics. In some classes of neural network mod- 
els, such as Boltzmann machine, the input-output is as- 
sumed to be stochastic. Once a stochastic neural net- 
work has converged to an equilibrium state, the proba- 
bility distribution characterizing ip is expected to obey 
the Boltzmann distribution 

m = exp(-SM) (21) 

z 

for the partition function 

Z = 5^exp(-£?[V]). (22) 

In neural processing architecture, the notion of entropy 
or free energy is put into practice ahead for the purpose 
of informatics. Compared to deterministic firing mod- 
els, an expected advantage in stochastic neural network 
models is to escape from poor locally optimal configu- 
rations through probabilistic evolution. Moreover, there 
are several reasons that the stochastic behavior should 
be indispensable process in neural networks. In view of 
learning rules, it is natural that neural states are occu- 
pied with features corresponding to frequent inputs (the 
coarse coding principle). On the other hand, it is effi- 
cient for a neural network to avoid the occupation with 
a few of features, so that an object is coded by a small 
population that is active for an event (the sparse coding 
principle). It is usual that the cost function in unsuper- 
vised learning algorithm is similar to the Helmholtz free 
energy that 

F = E-TS, (23) 
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where the parameter T is considered just as a positive 
constant that determines the importance of the second 
term relative to the first. In learning rules, the energy 
term is illustrated by a measurement how well the code 
describes the input data or carry the informations : 

E = (l/A0£E P ^|v)P(v|P) 

i v£V 

= (1/N)Y,P(HV) (24) 

i 

= 5^P(V»)P(^|2?), 

where a distribution P(ip\T>) is the average over the prob- 
ability that input v G T> generates output ip. In Hebbian 
development models, this energy term can be considered 
as an external source term, that is the average over the 
product between feature state and external signals, — Btp 
in a complex cell model (or —ipQ^ip in a simple cell 
model) as well. In learning rules, the entropy term as- 
sesses the sparseness of the code by assigning a cost de- 
pending on how the activity is distributed. According to 
Shannon's coding theorem, the amount of information is 
defined by 

S = -K^2P(ip)lnP(i/}) (25) 

where if is a positive constant and — \nP(ip) is the cost 
of the code, the number of bits required to communi- 
cate the code. The connections between information 
theory and statistical mechanics are rigorously investi- 
gated [i^l, |32, 03 . However, there is some hardship to 
apply the statistical mechanism to the phenomena in the 
real brain. Since neural process, comprehends the dy- 
namics at various spatial and temporal levels, is essen- 
tially dynamical and non equilibrium phenomenon. For 
example, the relaxation process in cortical map formation 
is very slow and the observed maps often do not satisfy 
the equilibrium criteria. The map formation in visual 
cortex occurs concentrately for several weeks or months 
after birth, during a so-called critical period. In observed 
orientation preference maps, the non-uniforming direc- 
tions of gradient (V^ii 7^ const, however |V0ii| ~ const 
for the longitudinal component 0||) and non- vanishing 
singular points (or pinwheels) indicate that the system 
may be frozen during the relaxation process Q . 

V. APPLICATION TO VISUAL MAP 
FORMATION MODELS 

According to the studies of the statistical structure of 
natural images, the response properties of visual neu- 
rons, the spatially localized and oriented, are consid- 
ered to be due to the efficient coding of natural im- 
ages |3 ll . Oriented bar or grid patterns are the most 
probable activity and the feature with 0(2) (or U(l)) 



symmetry components is a meaningful representation in 
orientation columns. With ocular dominance columns, 
the total feature can be expanded to 0(3) symmetry 
components with the restriction of synaptic normaliza- 
tion within each column. Therefore, the conventional 
spin vector (S x , S y , S z ) can serve as a useful representa- 
tion of the feature states with the preferred orientation 
4> = (1 /2)tan~ 1 (S x I S y ) . The proposed models of visual 
map formation are based on so various mechanisms. We 
rewrite four widely used visual map development models 
in term of FBM representation classified by their effective 
interaction terms that 

(A) Lateral interaction models : D = J 

(B) Recursive interaction models : D = (I — J) -1 

(C) Elastic-net model : D = J + S 

(D) SOFM algorithm : D = SJ. 

The lateral activity function J(ri,Tj) in the elastic- net 
model and the SOFM algorithm is taken to be all non- 
negative, the two-point interaction function D{ri,v.j) ~ 
D{\Yi — Yj\) takes the Mexican hat type for all cases ow- 
ing to the scattering funciton Sfa, rj). Periodic patterns, 
such as linear zones in orientation preference columns or 
parallel bands in ocular dominance columns, can develop 
when there are abundant negative values in D(ji,rj) so 
that D(q) in Fourier space has a non- vanishing minimum 
point q* with the wavelength A = 2ir/q*. 



A. Lateral Interaction Models 

A simple cell model of the visual map development 
uses the high-dimensional feature vector coding for the 
strength of the connection from each cortical location to 
each retinal (or LGN) location. The synaptic plasticity 
depends on the average over the activities of competing 
inputs, which are left and right eyes for ocular dominance 
columns or ON-center and OFF-center cells for orienta- 
tion preference columns. For a linear activation function 
or f{v) = v, the energy in Eq^H becomes that 

£[ w ] = -\ E E(*« + J ^S W » W ^- ( 26 ) 

hi a, (3 

This energy is decomposed with the (globally) trans- 
formed synaptic weights into the sum and difference : 

W + = W R + W L and W = W R - W L (27) 

for ocular dominance columns or similarly W± = 
Wojv± Woff for orientation columns. In a pixel-based 
representation for orientation columns, oriented patterns 
with low-frequency compose a dominant feature space 
and the energy is decomposed with the locally trans- 
formed weights as well. Therefore, the energy as the 
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function of field variables ip in a transformed and reduced 
feature space is that 

E bP] = - ^E^'^^^^i) (28) 

i,3 

for D = I + J. The input correlation matrix Q( 2 ) is ig- 
nored, because the frequency of the dominent features in 
inputs is regarded to be the same or the two-point activ- 
ity function D comprises this. The term — | ^ -0(r^) 2 , 
related to the self-relaxation term, does not effect in- 
fluence on the typical spacing of an emergent columnar 
pattern. In a complex cell model, the simplest is given 
by the summation of the neighbor interactions and the 
external stimuli terms as 

E W\ = -^E^'^te)^) - ^S(r i )V(r i )(29) 

i,j i 

for D = J. The external stimuli B{ri) is considered 
to be constant or vanishing. Therefore, the form of the 
lateral activity function J determines the typical appear- 
ance of developed feature map for both cases. In lateral 
interaction models, J is taken as the activation kernel, 
or Mexican hat function (positive feedback in the center, 
negative in the surroundings). For example, a well-known 
Mexican hat function, the called difference of Gaussians 
(DOG) filter, is described as 

J(r,,r 3 -) - e ( e -l*-r»l a /3o? „ ke -\'i-'i\ 3 ( 30 ) 

where k is the strength of inhibitory activity. Another 
example of Mexican hat function modified from a wavelet 
is given by 

J(r 4 , rj ) =s(l- fc kl ~ 2 rj|2 ) e^-r.l W ( 3i) 

for the lateral cooperation range 07. If the strength 
of inhibitory activity k is larger than threshold k c (= 
1/4), D (q) has a non- vanishing maximum point at q* = 
(1/(7)^4=17* 0. 



B. Recursive interaction models 

For a linear activation function, the output in Eq.JSJ) 
becomes 

Hi = V{ + E J ij V j + E J i3 J jkVk H , (32) 

j 3,k 

which is the summation of recursive recurrents. The en- 
ergy as the function of synaptic weights is obtained that 

£[W] = -\ E E A,QSw m W j73 , (33) 

i,j a, f3 



where the two-point interaction function is 

D = I + J + J 2 + --- = (I-J)- 1 (34) 

and the real parts of the eigenvalues of J are expected 
to be less than 1. Eq.(J2H|) is a simple modified equation 
of Miller's correlation-based learning models j3^. In the 
original representation by Miller et ai, the input stimuli 
term is described by an arbor function, expressing the lo- 
cation and the overall size of the receptive fields H, 0] . 
The two-point interaction function D takes the Mexican 
hat type and the wavelength of visual pattern is deter- 
mined by the peak of D(q) in the analysis by Miller as 
well |23. 

C. Elastic-Net Model 

The elastic-net model is described by an iterative pro- 
cedure with the update rule : 

A$(r 4 ) = a E (r t ) - *(r 3 )) 

|r, — rj I —a 

-|V-$( r< )| 2 /2 CT = 

+ /3(V-$(rO) s _| V _^ (rj) | W , (35) 

where a feature vector in the low-dimensional represen- 
tation is 

$(r) = (r x ,r y ,qsm(2cj)(r)),qcos(2(t>(r)),z(r)) 
= (r,V( r )) 

for the retinal location r = (r x ,r y ), the preferred orien- 
tation </>(r), the degree of preference for that orientation 
and the ocular dominance z [y, [12J • At each iteration, a 
stimulus vector V = (r v , v) is chosen at random accord- 
ing to a given probability distribution. The first term 
in Ea. l|35J) denotes the elastic force or the excitatory in- 
teractions between the nearest-neighbors, and the second 
term implies the normalized stimuli distributed around 
an activity center. Functional Taylor expansion of the 
right hand side after dropping all nonlinear terms leads 
to 

A^(r 4 )=a M*i)-#-i)} (36) 

\n — rj I —a 

-Mn) + E< v * v >M r *) " ^)) 

s j 

where the stimulus at position r^, 

v . = ve -\'i-rv\'/2cl (37) 

is distributed in a gaussian form with the activity center 
IV and the feedforward cooperation range a s . The corre- 
lation between the external stimuli at position and Yj 
is obtained by 

~ (ira 2 s /a?){v 2 ) v e-^- r ^/^. (38) 
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Therefore, the effective energy of the elastic-net model 
can be represented through the form in Eci. (|29[l . where 
the two-point interaction function is given by 

D = -/3I + J + S. (39) 

The lateral activity function becomes J(r^, Vj) — a8(\vi — 
Yj | — a) or the Laplacian operator in a continuum 
limit. The scattering function coincides with the form 
in Ea. H20|) for r\ = Pa 4 /8ira^ or is obtained by 

5(r, )rj O-A ( ^^..__^ e -l^lW). (40) 

This result means that the scattering function S(ri,rj) 
can act as an kernel with inhibitory activity however the 
lateral activity function J(ri,Tj) is nonnegative. There 
are also interaction terms of higher power but the two- 
point interaction function D(r il Yj) determines the major 
characteristics of developed feature maps. We transform 
it to Fourier space and obtain 

D(q) = -/3 + J(q) + 5(q) (41) 

It has a maximum at 

,* = lyC(|^), (42) 

which corresponds to the analytic results from different 
approaches fl3llil|. 



Moreover, the SOFM algorithm requires the hard com- 
petition, the called "winner take all" (WTA) case. As 
a s approaches zero (or large rf), the activity is local- 
ized only around the winning neuron and the scattering 
function in Fourier space becomes S(q) ~ (3(v 2 )-Dq 2 , the 
Laplacian operator. The lateral activity function in the 
SOFM approaches takes on the Gaussian form J(r^, Yj) = 
e -\Ti-T } \ /2a t for the lateral cooperation range <7; (lateral 
plasticity control) . Therefore we obtain the two-point in- 
teraction function 

£(q) = \s(q)J(q) = naf(3(v 2 ) v q 2 e^^l 2 (45) 

in Fourier space or 

D^rj) = f3(v 2 ) v (l - e-l-^l 2 /^ 2 (46) 

in real space. This is the Mexican hat function in Ea. l|31|) 
with k = 0.5. Ea. l45|) has a minimum at 

q* = V2/a u (47) 

which agrees with previous analytic results [Til ITsl l48l | 

and always positive if u\ > 0. The Kohonen's SOFM 
algorithm reads to robust learning rules because it al- 
ways succeeds in achieving an array of different feature 
detectors or a columnar pattern. 

VI. DISCUSSION 



D. Self-Organizing Feature Map Algorithm 

In Ea. (|29|l . the interaction term ipJip denotes the ex- 
change of spontaneous spikes, created without external 
activity. Spontaneous firings can occur in coupled non- 
linear oscillators with small dynamic fluctuations , w hich 
have been observed in some experiments |37J, |3g, |39j, |40j, 
However, several experiments suggested that the 
organization of feature maps is possible after the expo- 
sure to the external activity. In this case, the probability 
of spontaneous firing are small ( J <C v), so that the most 
intracellular interactions would be achieved by indirect 
currents of external activities. With the provoked inter- 
actions by external activities, we can take the effective 
energy as 

E W = - (E w + \ E (I E ^) ■ ( 43 ) 

If B(r)ip(r) is constant for all position r, the first term 
with ipjip supports the lateral interaction models again. 
In the Kohonen's SOFM algorithm, the lateral currents 
induced by feedforward normalized stimuli are focused 
and the effective interaction term is given by 

D{v u v 3 ) = i^S(r 4 ,r)J(r,r,). (44) 



The physical models of neural network based on neuro- 
science attempt to interpret both physiologic phenomena 
and computational architectures. In order to study the 
functional of the real brain, we need more adaptable the- 
ories than the basic neural architecture with connection- 
ism. In this paper, we show that the neural process at 
the cortical level can be described by using the conven- 
tional expressions in statistical physics. As we showed 
in visual map formations 0, , the collective neural dy- 
namics can be much alike well-known phenomena in the 
physical systems. 

In the assumption of neural network composed of 
columnar modules, we classify the synaptic connection 
types and anticipate different functional characters in 
computational processing. (1) In the connectivity be- 
tween close neurons within a columnar module, the func- 
tional attributes of neurons and the associative memory 
is realized. (2) By the connectivity between columnar 
modules within a cortex area, noted by the lateral activ- 
ity function or recurrent weight matrix J, the networks 
control laterally the output activity between neighbors. 
(3) Via the connectivity between far apart neurons cross 
cortex areas, neurons get driven-activity from external 
environment or other functional cortex areas. The colum- 
nar modules become elements (or nodes) again with high 
dimensional attributes in networks of neural networks. If 
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the recurrent weights matrix J is specified depending on 
the positions, the connectivity between columnar mod- 
ules also work in information coding. The connectivity 
between columnar modules within or beyond cortex areas 
would be strengthened also if there are much communi- 
cations between them according to the Hebbian rule, and 
there are some models holding the updating rule in the 
recurrent weights matrix J, such as the Goodall rule. [Zij] . 
We regard that the enhancement of connectivity between 
columnar modules proceed to the efficient communica- 
tions between neurons rather than information coding. 
The consideration of minicolumn as a columnar module 
and the processing element in network is optional. The 
formation of structure in minicolumn is also due to the 
functional grouping between neurons with similar inter- 
ests, and expected to be certified with more fundamental 
process at the cellular or molecular level. 

Extraction of the significant features in the input data 
is the purpose of an unsupervised learning rule and also 
expected to be a principle character of artificial and 
physiologic neural networks. The FBM representation 
method suggests how neurons find features from afferent 
signals and build knowledgement at the cortical level. An 
abstract representation of features in the FBM represen- 
tation and a symmetry breaking between feature com- 
ponents in progress is related to the learning process in 
the neural network. For example, difference looks of an 
object form a submanifold in pattern space and the pat- 
terns of the object can be abstracted and decomposed in 



the transformed and reduced feature space. 

In view of dynamics, the essential factors in neural 
process are (1) statistical structure of inputs, (2) attrac- 
tive or repulsive interactions between neighbor neurons, 
and (3) stochastic behavior of neurons. In this paper, we 
did not fully apply thermodynamic mechanics into neural 
process. There are some models which contain thermody- 
namic approach. The basic ingredients of Tanaka's Potts 
spin models are those of the lateral interaction models 
but he took a probabilistic evolution rather than a energy 
gradient flow ^.IHIiHiHiil. Piepenbrock presented a 
model which uses the effect of stochastic behavior in neu- 
ral network as a competition process |49j | . However there 
is no lateral inhibitory activity or feedforward competi- 
tion, thermodynamic effect can make a network to have a 
columnar structure with a thermal excitation at low tem- 
perature. We expect the stochastic behavior of neurons 
can be the connection between the physical neural dy- 
namic models and the neural network models originated 
from learning theory and an essential factor in compre- 
hension of systematic ordering-disordering or bifurcation 
problems in the real brain. Moreover, we expect that 
the theoretic experience in physics can offer more intu- 
itive appreciation of the physiologic phenomena at higher 
level and sophisticated mechanisms in computational ar- 
chitecture. 

This work was supported by the Ministry of Science 
and Technology and the Ministry of Education. 
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